[sandbox audit] Pin the sandbox server binary by digest and verify it before running it - #4837

[sandbox audit] — PR 8 of 9 in this repo's stack; merge in order.
Previous: #4836 · Next: #4835
Review only the commits this PR adds on top of its base; bases collapse to main as the stack lands.

Companion server PR: huggingface/sandbox-server#24 — these two have to land together, server first. See "Rollout".

What was wrong

Every sandbox job ran this at startup, as root, as PID 1, holding the derived sandbox token:

d=/tmp/.sbx-server
if command -v wget >/dev/null 2>&1; then wget -q -O "$d" "$SBX_SERVER_URL"
elif command -v curl >/dev/null 2>&1; then curl -fsSL -o "$d" "$SBX_SERVER_URL"
else cp "$SBX_SERVER_MOUNT/sbx-server" "$d"; fi
chmod +x "$d"
exec "$d"

$SBX_SERVER_URL was …/buckets/huggingface/sbx-server/resolve/sbx-server — the mutable alias that the server's publish workflow overwrites on every release. Nothing in the chain checked a version, a digest or a signature, so the client had no way to know which bytes it had just made PID 1 of a machine holding a user credential. A single write to that bucket path reaches every sandbox of every user on their next cold start.

Nothing suggests that happened. What bothers me is that the design already published the immutable artefact it needed (sbx-server-<commit>, right next to the alias) and simply didn't consume it.

Related, and it compounds the blast radius: wait_ready polled /health and read nothing out of it. A pool host keeps the binary it downloaded at boot for up to 24h, so a client and a server from different releases meet routinely — and the per-sandbox capability token from #4832 is exactly the kind of change that fails quietly under that, as a 403 on an unrelated route several calls later.

What changed

Pin. SANDBOX_SERVER_SHA256 and SANDBOX_SERVER_VERSION name one exact server build, next to DEFAULT_IMAGE. SBX_SERVER_URL now points at the digest-addressed object sbx-server-<sha256>, not the alias: a fetch that names the content it wants cannot be answered with different content.

Verify, and fail closed. The bootstrap checks the download against that digest and refuses on a mismatch. Specifics that matter:

• The check happens before chmod +x. An unverified file that is already executable is one slip away from being executed.
• sha256sum is not guaranteed in an arbitrary image, so openssl dgst -sha256 is the second try, and both feed the same string comparison so the two paths cannot drift.
• With neither tool the job refuses. SBX_ALLOW_UNVERIFIED_SERVER=1, passed through env=, is the only way past, and it can only skip a missing tool — never a failed comparison.
• The bucket-mount fallback (used by images with no wget or curl) reads the digest-named file and goes through the same check. It read the alias before; leaving it alone would have left the hole open on exactly the images least likely to notice.

Negotiate. wait_ready reads protocol from /health and raises a SandboxError naming host recycling when the server is older than this client needs. Permissive in one direction only: a newer server that still reports a protocol this client knows is accepted (it is declaring it kept serving it); an older one, or one that reports none at all, is refused. version is no use for this — it moves for a doc fix as readily as for a protocol break.

On failing closed

The plan was to fail closed and then measure the cost, so: nothing plausible lacks both hash tools.

python:3.12:       wget  curl  sha256sum  openssl
python:3.12-slim:  -wget -curl sha256sum  openssl
ubuntu:26.04:      -wget -curl sha256sum -openssl
alpine:3.22:       wget  -curl sha256sum -openssl
node:22:           wget  curl  sha256sum  openssl
busybox:latest:    wget  -curl sha256sum -openssl

Every one of them ships sha256sum. The default should cost nothing in practice; the escape hatch is for the image we haven't thought of.

Validation

$ PYTHONPATH=src python -m pytest tests/test_sandbox.py -q
64 passed

+9 cases: that the pin is a real sha256, that the URL names the digest rather than the alias, that verification precedes chmod +x, that the opt-out is reachable only after both tools are missing, and five on the protocol check (match, newer, older, absent, non-integer). ruff check and ruff format --check are clean; ty check src reports the same 4 pre-existing diagnostics as the base.

The bootstrap was also run by hand — the real musl binary, real images, a local python3 -m http.server, never the real bucket. Exec is detected by the server's own startup line; the good cases exit 0 after the idle watchdog fires, so "exec'd" and "refused" are never ambiguous.

A good case:

sbx-server 0.6.0 listening on 0.0.0.0:49912 (mode: dedicated, auth: required, landlock: abi 4 [fs,refer,truncate,no_tcp_bind])
sbx-server: idle for 5000ms, shutting down

A corrupted one — diagnosed, not silent, which matters because _tail_job_logs is what surfaces it inside the SandboxError:

sbx: sandbox server digest mismatch: got de7f74868aac777b7888b492258096de3cbe91f2b30a271a64fa1ccae4102609,
sbx: expected 501290eacb36a3bd8746b2f2ee20e85190ec3f697b4756c3e51a574384e61db4.
sbx: refusing to run it. Upgrade huggingface_hub if this client is pinned to a
sbx: digest that is no longer published.

And the no-tool refusal:

sbx: cannot verify the sandbox server digest: this image has neither sha256sum nor
sbx: openssl, so refusing to run it. To accept an unverified server, pass
sbx: env={'SBX_ALLOW_UNVERIFIED_SERVER': '1'} when creating the sandbox.

Not validated

• No live HF Jobs run. The digest path only truly exists inside a real job. One Sandbox.create() with the pinned digest, plus one with a deliberately wrong digest asserting the job fails fast with the digest error in its logs, is still owed before merge.
• The publish workflow has not been run either — see the server PR.

Rollout — do not merge this first

SANDBOX_SERVER_SHA256 currently holds the digest of a local musl build of the companion server branch. That object does not exist in the bucket, so as it stands every create() would fail closed with a 404 on the download. Correct behaviour, unshippable state. Order:

1. Merge the server PR.
2. Tag a server release; the workflow publishes sbx-server-<sha256> and prints the digest in its run summary.
3. Replace SANDBOX_SERVER_SHA256 here with that digest, re-run the live checks above, then merge.

Behaviour changes

• Every server release now needs a client release, and vice versa. That is the trade that makes the artefact verifiable; the protocol check is what makes the transition legible instead of mysterious.
• A job on an image with neither sha256sum nor openssl now fails to start unless SBX_ALLOW_UNVERIFIED_SERVER=1 is passed.
• The bootstrap script's text changed, and pool-host admission (from [sandbox audit] Validate a pool host before sending it a credential #4834) compares a candidate job's command against it — so hosts booted by an older client are no longer adopted, and the pool boots its own instead. Intended, and part of the same story as the protocol check.

Follow-ups, deliberately not folded in

• Signature verification. Once the server emits a build-provenance attestation or a cosign signature, the client should verify that and not just the digest. The digest is what closes the gap today because it is baked into a reviewed release.
• The protocol check only runs on wait_ready, which is only called on a job this client just booted. _connect_host and the pool-cache warm path build a transport without probing /health at all, so an adopted old host is currently caught by the command comparison above rather than by the protocol field. Extending the check to adoption means one extra request per candidate host, so it wants its own PR — happy to open the issue.
• The client could warn at create() time when the requested image is known to ship neither hash tool. The survey above suggests that set is empty, so I left it out.